Segmented customization

ABSTRACT

Techniques to facilitate provision of different user experiences to different groups of users of a mobile application are disclosed herein. In at least one implementation, a manifest provided by an application development and optimization platform is received that defines a plurality of user segments and a plurality of feature variants individually associated with the plurality of user segments. The manifest is processed to determine a segment of the plurality of user segments associated with a user of the mobile application along with a feature variant of the plurality of feature variants associated with the segment of the user. Execution of the mobile application is monitored for an occurrence of an event that triggers a default feature of the mobile application, and responsive to the event, the default feature is replaced with the feature variant associated with the segment of the user.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/968,020, entitled “SEGMENTED CUSTOMIZATION”, filed on May 1, 2018,which claims the benefit of, and priority to, U.S. Provisional PatentApplication No. 62/492,417, entitled “SEGMENTED CUSTOMIZATION” filed May1, 2017. Both applications are hereby incorporated by reference in theirentirety for all purposes.

TECHNICAL BACKGROUND

Modern computing systems such as smartphones, tablets, and other mobiledevices enable users to install and run various applications. In thefields of computer hardware and software technology, applicationdevelopers commonly use test features such as A/B testing, stagedrollouts, and feature toggling to test various features on subsets ofthe user base. For example, A/B testing, also called split testing,provides for simultaneously testing a modified version, or variant,against a pre-existing version by splitting the two versions betweendifferent user groups and monitoring the results. Staged rollout isanother testing technique (also referred to as “slow rollout” or “phasedrollout”), which involves pushing out a feature gradually to increasingnumbers of the user base. Feature toggling, or feature flagging, is atechnique where a new area of code or a newly designed feature iscompartmentalized to facilitate activation and deactivation of thefeature without affecting the rest of the application. The featuretoggling technique enables developers to push out partially tested codein a way where the code can be shut off quickly without a new buildrelease if any problems are detected. The above techniques can also becombined effectively. For example, once a feature is behind a flag, anapplication developer can expose it to a small subset of users to ensurea feature works as planned, and subsequently push the feature out to theremaining users gradually.

In various runtime environments, it is possible to modify how a method,function, class, or other such software component maps to the actualcode implementation of the component. Sometimes referred to asswizzling, such re-mapping technology allows code associated with onefunction to be replaced by code associated with another function. Thismay be useful in the context of making modifications to a user interfaceof an application, for example. Some implementations of re-mappingtechnology involve modifying the value of a pointer that points to alocation in memory where a method is implemented in code. By changingthe value of the pointer, the method can be made to point to other codesuch that, when the method is called, a different method is employed inits place. The target method that is called in place of the originalmethod is sometimes referred to as a callback function. Such techniquesmay enable a software development toolkit (SDK) to make modifications toan application without altering the main program code of theapplication. In other examples, the main program code could include boththe original method and the target method, allowing either method to becalled without utilizing the above re-mapping technology. For example,in a given A/B test, both variants could be coded into the applicationand then simultaneously tested between different groups of users.

Overview

Provided herein are techniques to facilitate provision of different userexperiences to different groups of users of a mobile application. In atleast one implementation, a manifest provided by an applicationdevelopment and optimization platform is received that defines aplurality of user segments and a plurality of feature variantsindividually associated with the plurality of user segments. Themanifest is processed to determine a segment of the plurality of usersegments associated with a user of the mobile application along with afeature variant of the plurality of feature variants associated with thesegment of the user. Execution of the mobile application is monitoredfor an occurrence of an event that triggers a default feature of themobile application, and responsive to the event, the default feature isreplaced with the feature variant associated with the segment of theuser.

This Overview is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. It may be understood that this Overview is not intended toidentify key features or essential features of the claimed subjectmatter, nor is it intended to be used to limit the scope of the claimedsubject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram that illustrates a communication system in anexemplary embodiment.

FIG. 2 is a flow diagram that illustrates an operation of acommunication system in an exemplary embodiment.

FIG. 3 illustrates an exemplary graphical display of an application on acomputing system.

FIG. 4 illustrates an exemplary graphical display of an application on amobile device for various user segments.

FIGS. 5-8 illustrate exemplary graphical displays of an application on acomputing system.

FIG. 9 illustrates exemplary graphical displays of an application on amobile device for various user segments.

FIGS. 10 and 11 illustrate exemplary graphical displays of anapplication on a computing system.

FIG. 12 is a block diagram that illustrates a computing system in anexemplary embodiment.

DETAILED DESCRIPTION

The following description and associated figures teach the best mode ofthe invention. For the purpose of teaching inventive principles, someconventional aspects of the best mode may be simplified or omitted. Thefollowing claims specify the scope of the invention. Note that someaspects of the best mode may not fall within the scope of the inventionas specified by the claims. Thus, those skilled in the art willappreciate variations from the best mode that fall within the scope ofthe invention. Those skilled in the art will appreciate that thefeatures described below can be combined in various ways to formmultiple variations of the invention. As a result, the invention is notlimited to the specific examples described below, but only by the claimsand their equivalents.

Mobile application designers often desire to make changes and updates tovisual elements and other aspects of the user interface of anapplication. Ordinarily, such changes would require the applicationdevelopers to edit program code to implement the new application designrequirements. However, a framework can be installed into a mobileapplication which can receive and interpret changes to visual propertiesof display elements, providing a quick and easy way for designers toedit the user interface of a mobile application without having to writeany programming code. Additionally or alternatively, various changes andfeature variants may be coded into the main program code of theapplication, thereby enabling code toggling to turn various features onor off. Such changes and new features can then be tested on subsets ofthe user base using various techniques such as A/B testing, stagedrollouts, and feature toggling. In some instances, these and othertechniques may also be utilized to implement customized segmentation ofusers and the provision of different changes and features to the varioususer segments to create different user experiences for each of thesegments. For example, the user base of any application may be segmentedby demographics such as age or gender, and a different user interfacevariant may be provided to a younger age group than an older segment.Such segmentation and customization of the user experience may beconfigured manually or could be determined programmatically. Forexample, an application developer could run an A/B experiment on allusers and the system could then analyze the results to automaticallyidentify that early adopters who have used the application for over oneyear perform better with one variant than new users who exhibit betterperformance with another variant.

Systems, methods, and software are disclosed herein that enhanceapplication development and software design-for-test (DFT) technologyutilizing an application development and optimization platform tofacilitate customized segmentation of users and providing different userexperiences to each of the segments. Among other benefits, thetechniques described herein provide application developers the abilityto manage how particular features are provided to various different usersegments. Additionally, the results of an A/B test may be evaluated andturned into permanent customizations per segment based on performanceanalysis. In some examples, the technology described herein canaccomplish these techniques by leveraging various embedded dynamic testfeatures such as, for example, dynamic variables and/or code blocks, orother similar programmatic dynamic test features. The followingdisclosure provides various techniques for creating and managingcustomizations per user segment.

Referring now to the drawings, FIG. 1 illustrates a communication systemthat may be utilized to implement customizations of visual elements andother features of an application per user segment. FIG. 2 illustrates anoperation of the communication system in an exemplary embodiment. FIGS.3-11 illustrate various exemplary graphical displays of an application,while FIG. 12 illustrates an exemplary computing system that may be usedto perform any of the techniques, processes, and operational scenariosdescribed herein.

Turning now to FIG. 1, a block diagram of communication system 100 isillustrated. Communication system 100 includes mobile device 101,computing system 103, communication network 130, and applicationmodification server 140. Mobile device 101 includes operating system 120and application 110. Application 110 runs on operating system 120.Mobile device 101 may also include a user interface that communicateswith operating system 120 over a bus communication device. Application110 comprises main program 111 and application modification softwaredevelopment kit (SDK) 112, which may be implemented as differentsoftware modules of application 110. Main program 111 comprises theprimary program instructions for the functionality of the application,such as streaming video, social networking, email, instant messaging,weather, navigation, or any other mobile application. Applicationmodification SDK 112 may be installed into application 110 to facilitatechanges and updates to a user interface and other visual elements of theapplication 110, perform A/B testing of different application designvariants, and other functionality. In some examples, applicationmodification SDK 112 could comprise an embedded control module ofapplication 110. Computing system 103 includes application editor 113.Computing system 103 may also include an operating system and userinterface, although these components are not shown for clarity.Application modification server 140 comprises a computing system thatprovides an application development and optimization platform. In someexamples, application editor 113 may comprise a web browser applicationthat loads the application development and optimization platformprovided by application modification server 140.

In operation, a developer of application 110 may execute applicationeditor 113 on computing system 103 to operate an application managementdashboard to apply real-time changes and updates to a user interface andother visual elements of the application 110, activate or deactivatefeatures for any segment of users, perform A/B testing of differentapplication design variants to determine how changes to application 110affect user behavior, customize the application 110 for different usersegments, and other functionality. The developer may execute application110 on mobile device 101 for use as a test device, and the execution ofapplication 110 is then mirrored in the visual editor 113 executing oncomputing system 103. The mirrored execution of application 110 withinapplication editor 113 is achieved by application modification SDK 112transferring screenshots of the application 110 to computing system 103for display within the editor 113, which may communicate over websockets. SDK 112 sends information about the user interface ofapplication 110 to computing system 103 for display within applicationeditor 113, including the entire view hierarchy of application 110,which comprises descriptions or labels of all views that exist in thecurrent interface and screenshots of the views. In this manner, thescreenshots of the views can be displayed as images on the screen withinthe visual application editor 113 and the view descriptions, labels, andany other information may be displayed in a tree structure or treediagram that represents the view hierarchy structure in a graphicalform.

Once the visual application editor 113 receives and displays the viewhierarchy of application 110, the developer can then click through thevarious views within the view hierarchy and make changes to differentvisual elements of the user interface. These changes are then sent tothe application modification server 140 which can instantly update thedisplay of application 110 with the changes in real-time on mobiledevice 101 via communication with application modification SDK 112.Similarly, other application management functionality of the visualapplication editor 113 may be created and communicated to applicationmodification server 140 and subsequently deployed to application 110 onmobile device 101 by communicating with SDK 112. Of course, any of thefunctionality described herein could be applied to numerous instances ofapplication 110 installed on multitudes of user mobile devices which mayaffect some or all of the entire user base, but only one mobile device101 is shown in FIG. 1 for clarity. An exemplary operation ofcommunication system 100 will now be discussed with respect to FIG. 2.

FIG. 2 is a flow diagram that illustrates an operation of communicationsystem 100 in an exemplary implementation. The operation 200 shown inFIG. 2 may also be referred to as customization process 200 herein. Thesteps of the operation are indicated below parenthetically. Thefollowing discussion of operation 200 will proceed with reference toelements of communication system 100 of FIG. 1 in order to illustrateits operations, but note that the details provided in FIG. 1 are merelyexemplary and not intended to limit the scope of process 200 to thespecific implementation of FIG. 1.

Operation 200 may be employed by computing system 101 to facilitateprovision of different user experiences to different groups of users ofmobile application 110. As shown in the operational flow of FIG. 2,mobile device 101 receives a manifest provided by an applicationdevelopment and optimization platform that defines a plurality of usersegments and a plurality of feature variants individually associatedwith the plurality of user segments (201). The manifest may be definedby an owner of application 110 or some other administrator with accessto the application development and optimization platform provided byapplication modification server 140. For example, the applicationdeveloper can configure the manifest to enable particular features forcertain user segments, such as demographic groups, user-definedsegments, groups of devices, and any other user segments. Additionallyor alternatively, the application developer can provide one or more newobjects, events, or actions with which to replace existing objects,events, or actions on a per-segment basis. The application developmentand optimization platform processes the modification instructions andupdates the corresponding manifest. In some examples, the manifest maybe requested by mobile device 101 and pulled from the applicationdevelopment and optimization platform provided by applicationmodification server 140, or could be pushed by server 140 automaticallyfor delivery to mobile device 101. For example, when mobile device 101launches application 110, a control module embedded into application110, such as application modification SDK 112 in this example, may querythe application development and optimization platform provided byapplication modification server 140 for the latest manifest. In otherwords, the manifest query may be communicated by mobile device 101responsive to launching or executing application 110. The applicationdevelopment and optimization platform would then responsively send themanifest (or updated manifest) for delivery to application 110 executingon mobile device 101. The manifest may be used to convey information tomobile device 101 about changes to mobile application 110 for varioususer segments. For example, the manifest can include instructions toenable or disable certain features per segment, instructions to replaceobjects with different objects upon occurrence of an action or event, orany other instructions. As will be discussed in greater detail below, insome instances the manifest can include an object or data that is toreplace an existing object upon occurrence of the action or event on aper-segment basis.

The plurality of user segments may be defined according to demographicinformation, such as age and gender, user role or skill level, such asnovice or advanced users, or any other categorization of groups ofusers. The user segments may also be user-defined, such as differentclasses of users defined by the owner of application 110, such asstandard users, green users, and gold users. Note that a particular usermay be a member of more than one user segment. In some examples, theplurality of user segments may be defined manually by an applicationdeveloper associated with mobile application 110, but may also bedefined automatically in some examples. For example, the applicationdevelopment and optimization platform may be configured to automaticallydetermine user segments from the results of certain A/B experiments orother variant testing, such as by identifying that users in a particulargeographic location exhibited improved performance with variant A overvariant B, whereas users in another location performed better withvariant B, and thus automatically identify these two user segmentshaving members in each of these geographic locations, respectively. Insome examples, the application development and optimization platform mayalso be configured to automatically define which of the plurality offeature variants are individually associated with the plurality of usersegments based on analysis of test results for the plurality of featurevariants over the plurality of user segments.

Mobile device 101 processes the manifest to determine a segment of theplurality of user segments associated with a user of the mobileapplication along with a feature variant of the plurality of featurevariants associated with the segment of the user (202). In someexamples, a control module embedded into application 110, such as SDK112, could process the manifest and determine the user segmentassociated with the user of application 110 along with the featurevariant associated with that user segment. Any objects, events, oractions specified in the manifest may optionally be stored in a localdata store on mobile device 101, which may later be accessed to replacea default feature with a feature variant as described below.Alternatively, in some instances, the manifest can simply enable ordisable features or code blocks of embedded feature variants on aper-segment basis. In such cases, the control module/SDK 112 can processthe manifest and enable or disable the features for the correspondinguser segment immediately, without waiting for an event trigger.

Mobile device 101 monitors execution of mobile application 110 for anoccurrence of an event that triggers a default feature of mobileapplication 110 (203). In some examples, the event that triggers thedefault feature and/or the default feature itself may be specified inthe manifest received by mobile device 101 and associated with thecorresponding feature variant for the user segment of the user.Alternatively, application 110 could be preconfigured to automaticallydetermine which event or events to monitor that are known to trigger thedefault feature that is associated with the feature variant specified inthe manifest for the user segment of the user. In some examples, acontrol module installed into application 110, such as SDK 112, maymonitor program instructions executing the native application 110 onmobile device 101 for occurrence of the event that triggers the defaultfeature of mobile application 110. In some examples, the event thattriggers the default feature could comprise a function call, user login,execution of a code block, or any other detectable programming event.

Responsive to the event, mobile device 101 replaces the default featurewith the feature variant associated with the segment of the user (204).As discussed above, in some instances the manifest can include objectdata that is to replace an existing object upon occurrence of an actionor event per user segment. When the event trigger occurs, application110 replaces the existing object with the object data specified in themanifest for the user segment associated with the user. For example,application 110 could replace the default feature with the featurevariant by replacing a default visual element with a variant visualelement associated with the segment of the user. In some examples, acontrol module installed into application 110, such as SDK 112, mayaccess a local data store to retrieve the object data provided in themanifest corresponding to the feature variant to replace the existingobject associated with the default feature. In this manner, the controlmodule, responsive to the event triggering the default feature, replacesthe default feature with the feature variant. In some embodiments,replacing the default feature with the feature variant associated withthe segment of the user comprises identifying an original function thatinvokes a function call associated with the event and responsivelyintercepting the function call and executing the feature variant insteadof the function call.

Advantageously, application 110 can receive and insert features into thedeployed application 110 by monitoring execution of the programinstructions and replacing code blocks, objects, user interfaces, andthe like with changed code blocks, objects, or user interface variantsfor a particular user segment. In this manner, operation of theapplication 110 can be modified on a per-segment basis without deployingdifferent versions to the different user segments and without modifyingthe object code of the deployed application. In addition, SDK 112 canmonitor feedback and/or otherwise track how the changes are received bythe various user segments and provide the tracking data to theapplication development and optimization platform. The development andoptimization platform can process the tracking data and provide usableresults and metrics to the application developers, allowing forreal-time dynamic feature deployments and tracking on a per-segmentbasis. Various operational scenarios involving mobile device 101 andcomputing system 103 for customizations per segment will now bedescribed with respect to FIGS. 3-11.

FIG. 3 illustrates an exemplary graphical display 301 of applicationeditor 113 on computing system 103. In this example, an applicationdeveloper is provided with an interface to manage segments associatedwith a mobile application, such as application 110 as shown in FIG. 1.Defining segments allows the application developer to easily evaluateexperiment results by segment and create optimized customizations forthe segments. In some examples, segments may be defined by deviceproperties, user demographics, and custom attributes, integrations withthird parties, bulk upload of user identifiers into a segment orassociating groups of user identifiers with different segments,conversion of test results into custom segments, and other segmentdefinitions. For example, a developer could run an A/B test on all usersof application 110 and visual editor 113 could analyze the results toidentify particular age groups of users that exhibit better performancewith different variants, and automatically define new segments based onthe identified age groups. Some examples of segments could includebusiness users and personal users, paid subscribers and unpaid users,veteran users and new users, or any other user segmentations. In theexample shown in FIG. 3 on graphical display 301, three user-definedsegments are shown as gold users, green users, and the rest.Additionally, users are also segmented by gender into male, female, andunknown, and into age groups of child, teen, adult, and senior. A givenuser could belong to multiple segment dimensions in some examples. Forexample, a given user could be a member of the gold user segment, themale gender segment, and the adult age segment. The applicationdeveloper is given the option to edit any of the segments through use ofthe edit buttons, or add new segment dimensions. In this manner, anynumber of custom segments may be defined and managed by the applicationdeveloper.

Referring now to FIG. 4, an exemplary graphical display 401 ofapplication 110 on mobile device 101 is illustrated. In this example,graphical display 401 provides a default home screen of application 110that is displayed for all users. In particular, as shown in FIG. 4,regardless of whether the user segment type is gold, green, or the rest,when these users log in to application 110, they are greeted by the samedefault home screen and the word “WELCOME” on a plain white backgroundas shown on graphical display 401. A technique to customize the homescreen displayed for each of these different user segments will now bedescribed with respect to FIG. 5-9.

FIG. 5 illustrates an exemplary graphical display 501 of applicationeditor 113 on computing system 103. In this example, an applicationdeveloper is provided with an application management dashboard to createvarious projects for a mobile application, such as application 110 asshown in FIG. 1. As shown in graphical display 501, the developer ispresented with the option to create a new project involving a featureflag to turn specific features on or off for any segment of users, anA/B experiment to determine how changes to a mobile application affectuser behavior, an instant update to make real-time changes to a mobileapplication, and a customization option to customize a mobileapplication for different segments. As shown in graphical display 501,in this example the application developer has selected the“Customization” option to customize application 110 for differentsegments. The result of selecting the “Customization” option is nextshown in graphical display 601A of FIG. 6.

FIG. 6 illustrates an exemplary graphical display 601A of applicationeditor 113 on computing system 103. Graphical display 601A is displayedas a result of the application developer selecting the “Customization”option shown in graphical display 501 to customize application 110 fordifferent segments as discussed above with respect to FIG. 5. In the“Customization” display shown in graphical display 601A, the applicationdeveloper is provided fields to enter the customization details in orderto name, describe, and tag the customization for easy reference. Afterentering the customization details, the user can select the desiredsegment dimension and edit the application for each segment. The link to“Manage Segments” would direct the user to the segment management screenas shown in graphical display 301 of FIG. 3 in order to edit existingsegments or add new segment dimensions as described above. As shown ingraphical display 601A, in this example the user has selected the“User-Defined Segments” segment dimension from the dropdown menu forwhich to apply the new customization. The “Customization” option screenis continued in graphical display 601B of FIG. 7.

FIG. 7 illustrates an exemplary graphical display 601B of applicationeditor 113 on computing system 103. Graphical display 601B is acontinuation of the “Customization” option screen of graphical display601A. Graphical display 601B provides the lower portion of the“Customization” option screen that the user began filling out as shownin graphical display 601A, where the user has scrolled the screen downto expose the content shown in graphical display 601B. Since the userhas selected the “User-Defined Segments” segment dimension from thedropdown menu as discussed above with respect to graphical display 601Aof FIG. 6, the user-defined segments of “Gold”, “Green” and the “Rest”are shown to provide the application developer the ability to editvarious elements of the application for each of these segments. In thisexample, the “Visual” option is selected, enabling the developer to viewand edit different visual elements of the application for each of thesegments. The developer may also select the “Code Block” and “DynamicVariables” options to edit these properties of the application for eachof the segments as well. As shown in graphical display 601B, in thisexample the user has edited the home screen for the “Gold” user segmentto display the text “WELCOME GOLD USER” and has changed the backgroundcolor to gold (as indicated by the lighter gray shading color), andedited the home screen for the “Green” user segment to display the text“WELCOME GREEN USER” and has changed the background color to green (asindicated by the darker gray shading color). The home screen for the“Rest” user segment remains unchanged as the default home screen withthe word “WELCOME” on a plain white background. Of course, any othervisual elements of the home screen could be edited in this manner foreach of the segments, including changing the arrangement of visualelements per segment, adding different text, buttons, images, video,navigation options, and any other elements per segment, or any otheredits to the visual elements for each of the segments. Further, anyother screens or views within the view hierarchy of the application 110could be selected and edited per segment in this manner, such as a loginscreen, main application screen, terms of service screen, and the like.Managing the selection of which version to display depending on the userproperties is a complex and difficult task, but is handled through theuser interface of application editor 113 in this example, which providesfor management of this complexity on the server side over a longer termor permanent basis, without having to code that selection complexityinto the application itself. When the application developer has finishedcustom editing the visual elements for each of the segments, the userclicks the “Launch Customization” button at the bottom of graphicaldisplay 601B. The result of selecting the “Launch Customization” buttonon graphical display 601B is shown in graphical display 601C of FIG. 8.

FIG. 8 illustrates an exemplary graphical display 601C of applicationeditor 113 on computing system 103. Graphical display 601C is displayedas a result of the application developer selecting the “LaunchCustomization” button shown in graphical display 601B to customizeapplication 110 for different segments as discussed above with respectto FIG. 7. As shown in graphical display 601C, a notification dialog boxis overlaid on top of the “Customization” option screen shown ingraphical display 601B of FIG. 7 as a result of the user selecting the“Launch Customization” button. The notification dialog box warns theuser that launching the customization will push the changes to each ofthe segments, and prompts the user to “Cancel” or “Continue”. In thisexample, the application developer selects the “Continue” button toapply the changes and deploy the customization to each of the segments.The result of the user launching the customization is shown in FIG. 9.

FIG. 9 illustrates the result of the application developer launching thecustomization as discussed above with respect to FIG. 8. In thisexample, each of the different user segments are greeted with customizedhome screens as defined by the application developer in thecustomization described above with respect to graphical display 601B ofFIG. 7. In particular, as shown in FIG. 9, after the “Gold” user segmentlogs in to the application, the “Gold” user is greeted with a customizedhome screen displaying the text “WELCOME GOLD USER” and a goldbackground color (as indicated by the lighter gray shading color).Similarly, after the “Green” user segment logs in to the application,the “Green” user is greeted with a customized home screen displaying thetext “WELCOME GREEN USER” and a green background color (as indicated bythe darker gray shading color). Finally, when the “Rest” user segmentlogs in to the application, the home screen for the “Rest” user segmentremains unchanged as the default home screen with the word “WELCOME” ona plain white background. In this manner, each of the different usersegments is provided a different experience when accessing application110 based on the customization defined by the application developer.

Referring now to FIG. 10, an exemplary graphical display 1001 ofapplication editor 113 on computing system 103 is illustrated. Graphicaldisplay 1001 provides an example of an interface to enable anapplication developer to create a segmented customization from theresults of running an A/B experiment. In this example, a specialonboarding flow experiment is run with two variants, A and B. Afteracquiring results, application editor 113 evaluates the data and canturn these results into a permanent customization per segment. Theuser-defined segment dimension having segments of “Gold”, “Green”, andthe “Rest” are displayed on graphical display 1001, along with theoption to “Customize”. The result of the application developer selectingthe “Customize” option is shown in graphical display 1101 of FIG. 11.

FIG. 11 illustrates an exemplary graphical display 1101 of applicationeditor 113 on computing system 103. Graphical display 1101 provides anexample of a display screen that may result from an applicationdeveloper selecting the “Customize” option as shown in graphical display1001 of FIG. 10 to create a segmented customization from the results ofrunning an A/B experiment. This option effectively stops the A/Bexperiment and turns it into a persistent customization optimized foreach user segment. In this example, the “Gold” user segment receives the“Original” variant, the “Green” user segment receives the“SpecialOnboardingFlowA” variant, and the “Rest” user segment receivesthe “SpecialOnboardingFlowB” variant. The application editor 113 maydetermine these segmented customizations automatically by analyzing theresults of the special onboarding flow experiment and recognizing thatthe assigned variants produced the most optimal results for theirrespective user segments. Accordingly, what was formerly an onboardingflow experiment becomes an onboarding flow customization. In thismanner, application editor 113 is able to automatically create asegmented customization from the results of running an A/B experiment,thereby greatly facilitating this process for application developers.

In addition, the segmented customization techniques disclosed herein mayenable anomaly detection, such as the ability to run a test anddetermine anomalies per segment or segment combination. For example, ifan application developer ran a test on an application comparing “Flow A”versus “Flow B” versus “Flow C” and found that “Flow B” was best formale users and “Flow C” for female users, the developer would beinclined to set up that long-term segmented customization. However,during this testing process it could be determine that while overall“Flow B” was best for male users, males in Europe had conversion drop tozero percent, while other geographic regions exhibited much betterconversion rates, indicating a major problem for the combination of“Flow B” for male users in Europe. Accordingly, a different variantcould be tested and deployed to male users in Europe instead of “FlowB”.

Now referring back to FIG. 1, mobile device 101 comprises a processingsystem and communication transceiver. Mobile device 101 may also includeother components such as a user interface, data storage system, andpower supply. Mobile device 101 may reside in a single device or may bedistributed across multiple devices. Examples of mobile device 101include mobile computing devices, such as cell phones, tablet computers,laptop computers, notebook computers, and gaming devices, as well as anyother type of mobile computing devices and any combination or variationthereof. Examples of mobile device 101 may also include desktopcomputers, server computers, and virtual machines, as well as any othertype of computing system, variation, or combination thereof.

Computing system 103 comprises a processing system and communicationtransceiver. Computing system 103 may also include other components suchas a user interface, data storage system, and power supply. Computingsystem 103 may reside in a single device or may be distributed acrossmultiple devices. Examples of computing system 103 include mobilecomputing devices, such as cell phones, tablet computers, laptopcomputers, notebook computers, and gaming devices, as well as any othertype of mobile computing devices and any combination or variationthereof. Examples of computing system 103 also include desktopcomputers, server computers, and virtual machines, as well as any othertype of computing system, variation, or combination thereof.

Communication network 130 could comprise multiple network elements suchas routers, gateways, telecommunication switches, servers, processingsystems, or other communication equipment and systems for providingcommunication and data services. In some examples, communication network130 could comprise wireless communication nodes, telephony switches,Internet routers, network gateways, computer systems, communicationlinks, or some other type of communication equipment, includingcombinations thereof. Communication network 130 may also compriseoptical networks, packet networks, local area networks (LAN),metropolitan area networks (MAN), wide area networks (WAN), or othernetwork topologies, equipment, or systems—including combinationsthereof. Communication network 130 may be configured to communicate overmetallic, wireless, or optical links. Communication network 130 may beconfigured to use time-division multiplexing (TDM), Internet Protocol(IP), Ethernet, optical networking, wireless protocols, communicationsignaling, or some other communication format, including combinationsthereof. In some examples, communication network 130 includes furtheraccess nodes and associated equipment for providing communicationservices to several computer systems across a large geographic region.

Application modification server 140 may be representative of anycomputing apparatus, system, or systems that may connect to anothercomputing system over a communication network. Application modificationserver 140 comprises a processing system and communication transceiver.Application modification server 140 may also include other componentssuch as a router, server, data storage system, and power supply.Application modification server 140 may reside in a single device or maybe distributed across multiple devices. Application modification server140 may be a discrete system or may be integrated within other systems,including other systems within communication system 100. Some examplesof application modification server 140 include desktop computers, servercomputers, cloud computing platforms, and virtual machines, as well asany other type of computing system, variation, or combination thereof.

Referring now to FIG. 12, a block diagram that illustrates computingsystem 1200 in an exemplary implementation is shown. Computing system1200 provides an example of mobile device 101, computing system 103,application modification server 140, or any computing system that may beused to execute customization process 200 or variations thereof,although such systems could use alternative configurations. Computingsystem 1200 includes processing system 1201, storage system 1203,software 1205, communication interface 1207, and user interface 1209.Software 1205 includes application 1206 which itself includescustomization process 200. Customization process 200 may optionally beimplemented separately from application 1206 as indicated by the dashedlines surrounding process 200 in FIG. 12.

Computing system 1200 may be representative of any computing apparatus,system, or systems on which application 1206 and customization process200 or variations thereof may be suitably implemented. Computing system1200 may reside in a single device or may be distributed across multipledevices. Examples of computing system 1200 include mobile computingdevices, such as cell phones, tablet computers, laptop computers,notebook computers, and gaming devices, as well as any other type ofmobile computing devices and any combination or variation thereof. Notethat the features and functionality of computing system 1200 may applyas well to desktop computers, server computers, and virtual machines, aswell as any other type of computing system, variation, or combinationthereof.

Computing system 1200 includes processing system 1201, storage system1203, software 1205, communication interface 1207, and user interface1209. Processing system 1201 is operatively coupled with storage system1203, communication interface 1207, and user interface 1209. Processingsystem 1201 loads and executes software 1205 from storage system 1203.When executed by computing system 1200 in general, and processing system1201 in particular, software 1205 directs computing system 1200 tooperate as described herein for each implementation or variationsthereof. Computing system 1200 may optionally include additionaldevices, features, or functionality not discussed herein for purposes ofbrevity.

Referring still to FIG. 12, processing system 1201 may comprise amicroprocessor and other circuitry that retrieves and executes software1205 from storage system 1203. Processing system 1201 may be implementedwithin a single processing device but may also be distributed acrossmultiple processing devices or sub-systems that cooperate in executingprogram instructions. Examples of processing system 1201 include generalpurpose central processing units, application specific processors, andlogic devices, as well as any other type of processing device,combinations, or variations thereof.

Storage system 1203 may comprise any computer-readable storage mediacapable of storing software 1205 and readable by processing system 1201.Storage system 1203 may include volatile and nonvolatile, removable andnon-removable media implemented in any method or technology for storageof information, such as computer readable instructions, data structures,program modules, or other data. Storage system 1203 may be implementedas a single storage device but may also be implemented across multiplestorage devices or sub-systems co-located or distributed relative toeach other. Storage system 1203 may comprise additional elements, suchas a controller, capable of communicating with processing system 1201.Examples of storage media include random-access memory, read-onlymemory, magnetic disks, optical disks, flash memory, virtual memory andnon-virtual memory, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and that may be accessed by aninstruction execution system, as well as any combination or variationthereof, or any other type of storage media. In no case is thecomputer-readable storage media a propagated signal.

In operation, in conjunction with user interface 1209, processing system1201 loads and executes portions of software 1205, such as customizationprocess 200, to facilitate provision of different user experiences todifferent groups of users of a mobile application as described herein.Software 1205 may be implemented in program instructions and among otherfunctions may, when executed by computing system 1200 in general orprocessing system 1201 in particular, direct computing system 1200 orprocessing system 1201 to receive a manifest provided by an applicationdevelopment and optimization platform that defines a plurality of usersegments and a plurality of feature variants individually associatedwith the plurality of user segments. Software 1205 may further directcomputing system 1200 or processing system 1201 to process the manifestto determine a segment of the plurality of user segments associated witha user of the mobile application along with a feature variant of theplurality of feature variants associated with the segment of the user.Additionally, software 1205 may direct computing system 1200 orprocessing system 1201 to monitor execution of the mobile applicationfor an occurrence of an event that triggers a default feature of themobile application. Finally, software 1205 may direct computing system1200 or processing system 1201 to, responsive to the event, replace thedefault feature with the feature variant associated with the segment ofthe user.

Software 1205 may include additional processes, programs, or components,such as operating system software or other application software.Examples of operating systems include Windows®, iOS®, and Android®, aswell as any other suitable operating system. Software 1205 may alsocomprise firmware or some other form of machine-readable processinginstructions executable by processing system 1201.

In general, software 1205 may, when loaded into processing system 1201and executed, transform computing system 1200 overall from ageneral-purpose computing system into a special-purpose computing systemcustomized to facilitate provision of different user experiences todifferent groups of users of a mobile application as described hereinfor each implementation or variations thereof. For example, encodingsoftware 1205 on storage system 1203 may transform the physicalstructure of storage system 1203. The specific transformation of thephysical structure may depend on various factors in differentimplementations of this description. Examples of such factors mayinclude, but are not limited to the technology used to implement thestorage media of storage system 1203 and whether the computer-readablestorage media are characterized as primary or secondary storage.

In some examples, if the computer-readable storage media are implementedas semiconductor-based memory, software 1205 may transform the physicalstate of the semiconductor memory when the program is encoded therein.For example, software 1205 may transform the state of transistors,capacitors, or other discrete circuit elements constituting thesemiconductor memory. A similar transformation may occur with respect tomagnetic or optical media. Other transformations of physical media arepossible without departing from the scope of the present description,with the foregoing examples provided only to facilitate this discussion.

It should be understood that computing system 1200 is generally intendedto represent a computing system with which software 1205 is deployed andexecuted in order to implement application 1206 and/or customizationprocess 200 to operate as described herein for each implementation (andvariations thereof). However, computing system 1200 may also representany computing system on which software 1205 may be staged and from wheresoftware 1205 may be distributed, transported, downloaded, or otherwiseprovided to yet another computing system for deployment and execution,or yet additional distribution. For example, computing system 1200 couldbe configured to deploy software 1205 over the internet to one or moreclient computing systems for execution thereon, such as in a cloud-baseddeployment scenario.

Communication interface 1207 may include communication connections anddevices that allow for communication between computing system 1200 andother computing systems (not shown) or services, over a communicationnetwork 1211 or collection of networks. In some implementations,communication interface 1207 receives dynamic data 1221 overcommunication network 1211. Examples of connections and devices thattogether allow for inter-system communication may include networkinterface cards, antennas, power amplifiers, RF circuitry, transceivers,and other communication circuitry. The aforementioned network,connections, and devices are well known and need not be discussed atlength here.

User interface 1209 may include a voice input device, a touch inputdevice for receiving a gesture from a user, a motion input device fordetecting non-touch gestures and other motions by a user, and othercomparable input devices and associated processing elements capable ofreceiving user input from a user. Output devices such as a display,speakers, haptic devices, and other types of output devices may also beincluded in user interface 1209. In some examples, user interface 1209could include a touch screen capable of displaying a graphical userinterface that also accepts user inputs via touches on its surface. Theaforementioned user input devices are well known in the art and need notbe discussed at length here. User interface 1209 may also includeassociated user interface software executable by processing system 1201in support of the various user input and output devices discussed above.Separately or in conjunction with each other and other hardware andsoftware elements, the user interface software and devices may provide agraphical user interface, a natural user interface, or any other kind ofuser interface. User interface 1209 may be omitted in someimplementations.

The functional block diagrams, operational sequences, and flow diagramsprovided in the Figures are representative of exemplary architectures,environments, and methodologies for performing novel aspects of thedisclosure. While, for purposes of simplicity of explanation, methodsincluded herein may be in the form of a functional diagram, operationalsequence, or flow diagram, and may be described as a series of acts, itis to be understood and appreciated that the methods are not limited bythe order of acts, as some acts may, in accordance therewith, occur in adifferent order and/or concurrently with other acts from that shown anddescribed herein. For example, those skilled in the art will understandand appreciate that a method could alternatively be represented as aseries of interrelated states or events, such as in a state diagram.Moreover, not all acts illustrated in a methodology may be required fora novel implementation.

The above description and associated figures teach the best mode of theinvention. The following claims specify the scope of the invention. Notethat some aspects of the best mode may not fall within the scope of theinvention as specified by the claims. Those skilled in the art willappreciate that the features described above can be combined in variousways to form multiple variations of the invention. As a result, theinvention is not limited to the specific embodiments described above,but only by the following claims and their equivalents.

The invention claimed is:
 1. A method to facilitate provision ofdifferent user experiences in applications in A/B testing, the methodcomprising: receiving, in the A/B testing involving a plurality ofinstances of the applications each installed on a computing device, amanifest that defines a plurality of user segments and a plurality offeature variants, the manifest originated from an application developerof the applications, each feature variant associated with at least oneof the user segments, wherein the plurality of user segments comprisecategorizations of different groups of users of the applications, thecategorizations of different groups of users are determined based onevents of the users taken in the applications, and the applications areassociated with a default feature; processing the manifest to determinethe plurality of user segments the plurality of feature variants; andconducting, based on the manifest, an instance of the A/B testing for auser of a particular instance of the applications, the A/B testingtracking performance of the particular instance of the applications withrespect to a feature variant, conducting the instance of the A/B testingcomprising: monitoring execution of the particular instance of theapplications for an occurrence of an event at the particular instance ofthe applications, the event associated with how the categorizations ofdifferent groups of users are determined, determining that the userassociated with the particular instance of the applications belongs toone of the user segments; enabling, responsive to determining that theuser associated with the particular instance of the applicationsbelonging to one of the user segments, a changed code block instead of adefault code block to be executed by the particular instance of theapplications, the default code block corresponding to the defaultfeature and the changed code block corresponding to the feature variantassociated with said one of the user segments of the user, the defaultcode block and the changed code block both being part of the particularinstance of the applications at a time the particular instance of theapplications is installed on the computing device associated with theuser, triggering, at the particular instance of the applications, thefeature variant associated with said one of the user segments of theuser instead of the default feature by executing the changed code block,and transmitting data tracking the performance of the particularinstance of the applications as outputs of the instance of the A/Btesting.
 2. The method of claim 1, wherein triggering the featurevariant comprises replacing a default visual element with a variantvisual element.
 3. The method of claim 1, wherein the feature variantsassociated with which user segments are automatically defined based onanalysis of test results for the plurality of feature variants over theplurality of user segments.
 4. The method of claim 1, wherein theplurality of user segments are defined partially based on criteria setforth by the application developer associated with the applications. 5.The method of claim 1, wherein the event that triggers the defaultfeature comprises a function call.
 6. The method of claim 5, whereintriggering the feature variant associated with said one of the usersegments of the user comprises identifying an original function thatinvokes the function call and responsively intercepting the functioncall and executing the feature variant instead of the function call. 7.The method of claim 1, wherein the event that triggers the defaultfeature comprises a user login.
 8. The method of claim 1, wherein thechanged code block includes a dynamic variable.
 9. One or morenon-transitory computer-readable storage media having instructionsstored thereon to facilitate provision of different user experiences inapplications in A/B testing, wherein the instructions, when executed byone or more processors, cause the one or more processors to at least:receive, in the A/B testing involving a plurality of instances ofapplications each installed on a computing device, a manifest thatdefines a plurality of user segments and a plurality of featurevariants, the manifest originated from an application developer of theapplications, each feature variant associated with at least one of theuser segments, wherein the plurality of user segments comprisecategorizations of different groups of users of the applications, thecategorizations of different groups of users are determined based onevents of the users taken in the applications, and the applications areassociated with a default feature; process the manifest to determine theplurality of user segments the plurality of feature variants; andconduct, based on the manifest, an instance of the A/B testing for auser of a particular instance of the applications, the A/B testingtracking performance of the particular instance of the applications withrespect to a feature variant, wherein the instruction to conduct theinstance of the A/B testing comprises instructions for: monitoringexecution of the particular instance of the applications for anoccurrence of an event at the particular instance of the applications,the event associated with how the categorizations of different groups ofusers are determined, determining that the user associated with theparticular instance of the applications belongs to one of the usersegments; enabling, responsive to determining that the user associatedwith the particular instance of the applications belonging to one of theuser segments, a changed code block instead of a default code block tobe executed by the particular instance of the applications, the defaultcode block corresponding to the default feature and the changed codeblock corresponding to the feature variant associated with said one ofthe user segments of the user, the default code block and the changedcode block both being part of the particular instance of theapplications at a time the particular instance of the applications isinstalled on the computing device associated with the user, triggering,at the particular instance of the applications, the feature variantassociated with said one of the user segments of the user instead of thedefault feature by executing the changed code block, and transmittingdata tracking the performance of the particular instance of theapplications as outputs of the instance of the A/B testing.
 10. The oneor more computer-readable storage media of claim 9 wherein theinstructions that cause the one or more processors to trigger thefeature variant include instructions to replace a default visual elementwith a variant visual element.
 11. The one or more computer-readablestorage media of claim 9, wherein the feature variants associated withwhich user segments are automatically defined based on analysis of testresults for the plurality of feature variants over the plurality of usersegments.
 12. The one or more computer-readable storage media of claim9, wherein the plurality of user segments are defined partially based oncriteria set forth by the application developer associated with theapplications.
 13. The one or more computer-readable storage media ofclaim 9, wherein the event that triggers the default feature comprises afunction call.
 14. The one or more computer-readable storage media ofclaim 13, wherein the instructions that cause the one or more processorsto trigger the feature variant associated with said one of the usersegments of the user include instructions to identify an originalfunction that invokes the function call and responsively intercept thefunction call and execute the feature variant instead of the functioncall.
 15. The one or more computer-readable storage media of claim 9,wherein the event that triggers the default feature comprises a userlogin.
 16. A system comprising: one or more processors; memoryconfigured to store instructions, the instructions, when executed by theone or more processors, cause the one or more processors to perform atleast: receive, in an A/B testing involving a plurality of instances ofapplications each installed on a computing device, a manifest thatdefines a plurality of user segments and a plurality of featurevariants, the manifest originated from an application developer of theapplications, each feature variant associated with at least one of theuser segments, wherein the plurality of user segments comprisecategorizations of different groups of users of the applications, thecategorizations of different groups of users are determined based onevents of the users taken in the applications, and the applications areassociated with a default feature; process the manifest to determine theplurality of user segments the plurality of feature variants; andconduct, based on the manifest, an instance of the A/B testing for auser of a particular instance of the applications, the A/B testingtracking performance of the particular instance of the applications withrespect to a feature variant, wherein the instruction to conduct theinstance of the A/B testing comprises instructions for: monitoringexecution of the particular instance of the applications for anoccurrence of an event at the particular instance of the applications,the event associated with how the categorizations of different groups ofusers are determined, determining that the user associated with theparticular instance of the applications belongs to one of the usersegments; enabling, responsive to determining that the user associatedwith the particular instance of the applications belonging to one of theuser segments, a changed code block instead of a default code block tobe executed by the particular instance of the applications, the defaultcode block corresponding to the default feature and the changed codeblock corresponding to the feature variant associated with said one ofthe user segments of the user, the default code block and the changedcode block both being part of the particular instance of theapplications at a time the particular instance of the applications isinstalled on the computing device associated with the user, triggering,at the particular instance of the applications, the feature variantassociated with said one of the user segments of the user instead of thedefault feature by executing the changed code block, and transmittingdata tracking the performance of the particular instance of theapplications as outputs of the instance of the A/B testing.
 17. Thesystem of claim 16, wherein the instructions direct the system totrigger the feature variant by directing the system to replace a defaultvisual element with a variant visual element.
 18. The system of claim16, wherein the feature variants associated with which user segments areautomatically defined based on analysis of test results for theplurality of feature variants over the plurality of user segments. 19.The system of claim 16, wherein the plurality of user segments aredefined partially based on criteria set forth by the applicationdeveloper associated with the applications.
 20. The system of claim 16,wherein the event that triggers the default feature comprises a functioncall.